Using Latent Semantic Analysis to Grade Brief Summaries: 2

A study exploring texts at different academic levels. Abstract 26 27 In this study we propose an integrated method to automatically evaluate very brief 28 summaries (around 50 words) using the computational tool Latent Semantic Analysis 29 (LSA). The method proposed is based on a regression equation calculated with a corpus 30 of a hundred summaries (the training sample), and is validated on a different sample of 31 summaries (validation sample). The equation incorporates two parameters extracted 32 from LSA: (1) the semantic similarity of the summary, measured using the Summary– 33 expert summaries method (Landauer et al. The study is based on a sample of 786 35 summaries by students at four academic levels. All of these students summarized either 36 an expository or a narrative text; their summaries were then evaluated by four graders 37 on a scale of 0-10. The results support three ideas. First, that incorporating both 38 parameters into the method is more successful than the traditional cosine measure. The 39 reliability of LSA for evaluating summaries rises above the 0.80 level for the expository 40 text. Second, that LSA shows practically the same level of sensitivity as the human 41 graders to the quality of the summaries at different academic levels. Third, that the 42 method overcomes a serious limitation of LSA: its difficulties evaluating very brief 43